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ABSTRACT 

Objective: A comprehensive systematic review of 
economic evaluations of complementary and integrative 
medicine (CIM) to establish the value of these 
therapies to health reform efforts. 
Data sources: PubMed, CINAHL, AMED, Psychlnfo, 
Web of Science and EMBASE were searched from 
inception through 2010. In addition, bibliographies of 
found articles and reviews were searched, and key 
researchers were contacted. 
Eligibility criteria for selecting studies: Studies 
of CIM were identified using criteria based on those of 
the Cochrane complementary and alternative medicine 
group. All studies of CIM reporting economic 
outcomes were included. 

Study appraisal methods: All recent (and likely 
most cost-relevant) full economic evaluations 
published 2001-2010 were subjected to several 
measures of quality. Detailed results of higher-quality 
studies are reported. 

Results: A total of 338 economic evaluations of CIM 
were identified, of which 204, covering a wide variety 
of CIM for different populations, were published 
2001-2010. A total of 114 of these were full economic 
evaluations. And 90% of these articles covered studies 
of single CIM therapies and only one compared usual 
care to usual care plus access to multiple licensed CIM 
practitioners. Of the recent full evaluations, 31 (27%) 
met five study-quality criteria, and 22 of these also met 
the minimum criterion for study transferability 
('generalisability'). Of the 56 comparisons made in the 
higher-quality studies, 16 (29%) show a health 
improvement with cost savings for the CIM therapy 
versus usual care. Study quality of the cost-utility 
analyses (CUAs) of CIM was generally comparable 
to that seen in CUAs across all medicine according to 
several measures, and the quality of the cost-saving 
studies was slightly, but not significantly, lower 
than those showing cost increases (85% vs 88%, 
p=0.460). 

Conclusions: This comprehensive review identified 
many CIM economic evaluations missed by previous 
reviews and emerging evidence of cost-effectiveness 
and possible cost savings in at least a few clinical 
populations. Recommendations are made for future 
studies. 



ARTICLE SUMMARY 



Article focus 

■ Given the limited nature of previous systematic 
reviews, what is the extent of evidence on the 
economic impacts of complementary and inte- 
grative medicine (CIM)? 

■ What are the range of therapies and populations 
studied, and the quality of published economic 
evaluations of CIM? 

■ What are the results of the higher-quality, more 
recent (and likely most cost-relevant) economic 
evaluations of CIM? 

Key messages 

■ This study's comprehensive search strategy iden 
tified 338 economic evaluations of CIM, includ 
ing 114 full evaluations published 2001-2010. 

■ The cost-utility analyses found were of similar or 
better quality to those published across all medicine. 

■ The higher-quality studies indicate potential cost- 
effectiveness, and even cost savings across a 
number of CIM therapies and populations. 

Strengths and limitations of this study 

■ The strengths of this study are the comprehen 
sive search strategy, the use of two reviewers 
the use of multiple measures of study quality 
and the identification of higher-quality studies 
for which results are reported in detail, via an 
objective short-list of quality criteria, which 
reduced the potential for bias. 

■ The weaknesses of this study are similar to 
those of the other systematic reviews: reviewers 
were not blinded to journals and article authors 
and some aspects of what makes a quality eco- 
nomic evaluation could not be judged from what 
was reported. 

■ Publication bias was not assessed. However, it is 
not clear as to whether publication bias is rele- 
vant, given the purposes of this review. 



INTRODUCTION 

Between 1990 and 2007, four nationally 
representative surveys demonstrated that a 
third or more of US adults routinely used 



Herman PM, Poindexter BL, Witt CM, etal. BMJ Open 201 2;2:e001 046. doi:1 0.1 1 36/bmjopen-201 2-001 046 



1 



Economics of complementary and integrative medicine 



complementary and alternative medicine (CAM) therap- 
ies to treat their principal medical conditions. 1-4 Total 
expenditures for CAM therapies were estimated at US 
$14 billion in 1990, 1 US$27 billion in 1997 2 and US$34 
billion in 2007. 4 The 2007 US National Health Inventory 
Survey found that out-of-pocket expenditures for CAM 
therapies accounted for 11% of all out-of-pocket health- 
care expenditures by Americans. 4 Similar use numbers 
are seen in other countries. 5-8 However, despite the 
popularity of and substantial expenditures on CAM ther- 
apies, their cost-effectiveness remains ill-defined and 
controversial. 

Economic evaluations allow costs to be included, 
alongside data on safety and effectiveness, in healthcare 
policy decisions. As healthcare costs rise, the availability 
of these economic evaluations becomes increasingly 
important to the formulation of disease management 
strategies which are both clinically effective and finan- 
cially responsible. According to the National Center for 
Complementary and Alternative Medicine (NCCAM), 
CAM is 'a group of diverse medical and healthcare 
systems, practices and products that are not generally 
considered part of conventional medicine'. 9 In not 
being part of conventional medicine, individual comple- 
mentary therapies and emerging models of integrative 
medicine (ie, coordinated access to both conventional 
and complementary care) — collectively termed as com- 
plementary and integrative medicine (CIM) — are often 
excluded in financial mechanisms commonly available 
for conventional medicine, 2 and are rarely included in 
the range of options considered in the formation of 
healthcare policy. The availability of economic data 
could improve the consideration and appropriate inclu- 
sion of CIM in strategies to lower overall healthcare 
costs. In addition, economic outcomes are relevant to 
the licensure and scope of practice of practitioners, 
industry investment decisions (eg, the business case for 
integrative medicine), consumers and future research 
efforts (ie, through identifying decision-critical para- 
meters for additional research 10 ). 

A number of systematic reviews of economic evalua- 
tions of CIM have been published. 11-23 Five of these 
prior reviews attempted to capture all economic evalua- 

22 29 21 23 

tions of CIM therapies across all conditions. 
However, it is unclear as to whether all or even the 
majority of economic evaluations of CIM have been 
identified by these reviews. The searches are dated; the 
search strategy in the most recent review only captured 
articles published through 2007. 23 The databases 
searched were limited — for example, only one used 
CINAHL, 21 and only two others used EMBASE, 19 23 in 
addition to Medline and AMED. Finally, these reviews 
generally used limited search terms to identify CIM 
studies. All but one only used variations on the terms 
'complementary' or 'alternative' 'medicine' or 'therapy'. 
Unfortunately, other reviewers have found that these 
search terms do not capture all CIM studies, 24 25 which 
may be a reflection of the difficulty in defining what is 



and is not CIM. The search by Maxion-Bergemann 
et al 11 also added individual therapies as search terms, 
but only included homeopathy, phytotherapy, traditional 
Chinese medicine, anthroposophic medicine and neural 
therapy. No search included 'integrative medicine'. 

The goal of this paper is to identify, to the extent pos- 
sible, all published economic evaluations of CIM, 
describe the types of CIM evaluated and the clinical con- 
ditions for which they have been evaluated, and identify 
the recent (and therefore, most cost-relevant) higher- 
quality studies and highlight their results for policy 
makers. We also make recommendations for future eco- 
nomic evaluations of CIM. 



METHODS 

Six electronic databases were searched from their incep- 
tion through December 2010: PubMed, CINAHL, 
AMED, Psychlnfo, Web of Science and EMBASE. To be 
as comprehensive as possible, a combination of 11 
medical subject headings (MeSH) and 39 other search 
terms were used (box 1). In addition, bibliographies of 
found articles and reviews were searched, and key 
researchers in various areas of CIM were contacted for 
their lists of known studies. Although non-English lan- 
guage articles were collected, they are not analysed in 
this review. 

Defining a comprehensive search strategy for CIM is 
challenging. 24 27-29 There have been a number of efforts 
to develop a concise definition of CAM. 26 30 This review 
used the one developed by the members of the 
Cochrane CAM Field 31 and then added the terms 'inte- 
grative', 'integrated' and 'collaborative' medicine. The 
Cochrane CAM definition starts with the NCCAM defin- 
ition 9 and then refines it by specifically including all 



Box1 



Search terms used for the PubMed search: (Complementary 
Therapies (medical subject headings (MeSH)), Dietary Supplements 
(MeSH), Micronutrients (MeSH), Trace Elements (MeSH), Vitamins 
(MeSH), acupuncture, alternative medicine, ayurvedic medicine, 
chiropractic, biofeedback, collaborative medicine, complementary 
and alternative medicine, botanical medicine, complementary medi- 
cine, diet, energy medicine, herbal medicine, herbs, homeopathy, 
hypnosis, integrated medicine, integrative medicine, massage, 
meditation, mind-body medicine, minerals, naturopathic medicine, 
naturopathy, nutrients, nutritional supplements, relaxation, spa 
therapy, traditional Chinese medicine OR vitamins) AND (Cost- 
Benefit Analysis (MeSH), Cost Control (MeSH), Cost Savings 
(MeSH), Costs and Cost Analysis (MeSH), Economics (MeSH), eco- 
nomics (Subheading), Insurance (MeSH), cost benefit, cost effect- 
iveness, cost identification, cost minimisation, cost utility, economic 
evaluation, insurance claims, managed care OR technology assess- 
ment). Searches in the other five databases used the same text 
words and (where available) analogous controlled vocabulary terms. 
All searches were restricted to human studies. 
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therapies 'based upon the theories of a medical system 
outside the Western allopathic medical model' (eg, trad- 
itional Chinese medicine and Reiki), and including 
others depending on the context and setting of their 
use. The context of use considers treatment/condition 
combinations and excludes those 'currently considered 
to be standard treatment', and the setting of use gener- 
ally includes self-care and therapies delivered by CIM 
providers, but excludes therapies 'delivered exclusively 
by conventionally credentialed medical personnel or 
exclusively within hospital settings'. Therefore, therap- 
ies such as chemotherapy regimens (eg, chronother- 
apy 32 ), and therapies requiring surgical implantation 
(eg, neuroreflexotherapy 33 ) or the placement of a 
feeding tube 34 were not included even though these 
therapies appeared in our search. In cases where CIM 
therapies (eg, biofeedback or hypnosis) were included 
as part of a package of care (eg, with cognitive behav- 
ioural therapy) , a judgement was made as to whether 
the CIM portion of the treatment made up half or 
more of the overall package of care under study. If so, 
the package of care was included as CIM. Because 
more than half of CIM users use multiple CIM therap- 
ies, 35 studies of packages of therapies and coordinated 
care were identified as such. 

Articles were categorised as full economic evaluations 
if they compared the costs (inputs) and consequences 
(economic, clinical and/or humanistic outcomes 36 ) of 
two or more therapeutic alternatives applied to the same 
patient population (ref. 37 , p. 11). Otherwise, they were 
considered partial evaluations, for example, cost- 
identification or cost-comparison studies. 38 Studies that 
estimated resource utilisation were included as eco- 
nomic evaluations if the utilisation data were detailed 
enough to allow monetary valuation. 

Two reviewers (PMH and BLP) evaluated all articles 
for inclusion and extracted all data. Disagreements were 
resolved by discussion between the two review authors, 
or, if needed, by the other coauthors. Because the 
results of economic evaluations can rapidly lose rele- 
vance with time, mainly due to changes in practice pat- 
terns and cost structures, data were extracted only from 
the economic evaluations published 2001-2010. 
Extracted data were entered into an Excel template 
developed for a previous review. 20 The type(s) of CIM 
evaluated and the target population were captured for 
all economic evaluations. Various indicators of study 
quality were captured for all full economic evaluations, 
and more detailed data and results were captured only 
for those full economic evaluations that met five quality 
criteria. 

The quality of an economic evaluation can be judged 
along two general dimensions: (1) whether the study was 
a quality measure of outcomes for its target population 
and location — that is, whether it was internally valid; and 
(2) whether enough information is provided for the 
study's results to be transferable ('generalisable'). 39 
Health outcomes are to some extent considered 



generalisable across settings; however, because resource 
availability, practice patterns and relative prices can vary 
greatly, economic outcomes usually are not. 40 Therefore, 
one goal in economic evaluation is to ensure the transfer- 
ability of study results — that is, to provide enough study 
detail so that results can be adapted (usually via model- 
ling) to apply to other settings. 39 The 35-item BMJ check- 
list captures components of both dimensions of quality 
and was applied to all full economic evaluations. 41 We 
also chose five quality criteria by which to identify a 
subset of full economic evaluations to highlight as being 
of most interest to policy makers. These quality criteria 
are based on recommendations made by the US Panel on 
Cost Effectiveness in Health and Medicine 42 and by well- 
known experts in the field, 37 and focus on the quality of 
the underlying study (the first type of quality): 

► Because cost-effectiveness analysis (CEA) is compara- 
tive, to ensure that results are useful to decision 
makers, one of the alternatives to which the CIM 
intervention was compared must be some version of 
commonly available (routine, standard or usual) care. 

► The analysis must explicitly or implicitly use (and 
include all relevant costs from) at least one recog- 
nised perspective — for example, society, third-party 
payer, hospital or employer. 

► Since 'an economic evaluation of a healthcare pro- 
gramme is only as good as the effectiveness data it is 
built upon' (ref. 43 , p. 232), health outcomes must be 
from randomised controlled trials or non-randomised 
controlled trials using either statistical adjustment or 
matching to address baseline differences. 

► Since having patient-specific data on both costs and 
outcomes is an advantage for internal validity, 44 
resource use must be a measured outcome of the 
study. Modelling studies utilise the results of other 
published studies, therefore, are exempt from this 
criterion. 

► Because uncertainty in an economic evaluation 
comes not just from sample variation, but also from 
assumptions made, 45 a sensitivity analysis is required. 

Because the prices used to value resources are highly 
location-specific and setting-specific, 39 46 we also note, 
for the articles meeting the above criteria, the presence 
of a study reporting criterion essential for the transfer- 
ability of study results (usually via modelling): 39 40 separ- 
ate reporting of unit costs from resource use for 
economic evaluations alongside trials, or from model 
parameters (eg, transition probabilities) for economic 
evaluations using models. 

Other data extracted for the economic evaluations 
which meet the five study-quality criteria are: treatment 
and study duration, primary clinical and economic 
outcome measures, the setting in which treatment took 
place, study design and sample size, the type (table 1) 
and perspective (ie, the point of view used to define 
costs) of the economic analysis, and incremental cost- 
effectiveness of the CIM alternative compared to 
usual care. Incremental cost-effectiveness is reported in 
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Table 1 Types of full economic evaluations 




Cost-utility analysis 

Cost-benefit analysis Cost-effectiveness analysis (CEA) (a special case of CEA) 

Unit of health outcome Monetary units (eg, US$) Natural units (eg, life-years gained) Units of overall impact on length 

and quality of life (eg, QALY) 

Results Net benefits Incremental cost-effectiveness ratio* Incremental cost-utility ratio* 
(B 1 - B 2 )-(d -C 2 -S 1 +S 2 ) (d -C 2 -S 1 +S 2 )/(E 1 - E 2 ) (C 1 -C 2 -S 1 +S 2 )/(QALY 1 -QALY 2 ) 

*Ratios are calculated when both the costs and the effects (health improvements) of one therapy alternative are higher than those of another. 
When the costs are lower and the effects are better for one therapy, it is said to dominate the alternative (and the alternative is said to be 
dominated) and no ratio is presented. B-,, monetary value of health outcomes of alternative 1 ; B 2 , monetary value of health outcomes of 
alternative 2; total input costs of alternative 1 ; C 2 , total input costs of alternative 2; S-i, total cost savings (economic outcomes) for 
alternative 1 ; S 2 , total cost savings (economic outcomes) for alternative 2; E-,, health effects of alternative 1 ; E 2 , health effects of alternative 2; 
QALY^ quality-adjusted life-years of alternative 1; QALY 2 , quality-adjusted life-years of alternative 2. 



2011 US$ and is calculated from reported results by first 
converting the study currency to US$ using the Federal 
Reserve annual exchange rate 47 for the study's currency 
year and then inflated to 2011 values using the medical 
care component of the Consumer Price Index. 48 

Finally, up to three additional quality measures are 
included for each of these studies. The Tufts CEA 
Registry 49 quality score is recorded when it was available 
(note it is only available for cost-utility analyses, CUAs). 
A Jadad score 50 with minor modifications (the two pos- 
sible points for blinding were replaced with one point 



for the use of a blinded assessor) 51 was calculated for 
the economic evaluations that included a randomised 
trial. The percentage of the applicable items from the 
35-item BMJ checklist that were met by each article is 
also reported. 41 



RESULTS 

As shown in figure 1, the database search identified 270 
published economic evaluations. An additional 68 arti- 
cles were added through the bibliography and expert- 



Figure 1 The flow of records 
and articles through the 
systematic review. 



PRISMA 2009 Flow Diagram 



Records identified through 
database searching 
{n = 39,377) 



Records after duplicates removed 
(n - 13,142) 



Articles from experts' lists 
and bibliographies 
(n = 68) 



Records (abstracts) 
screened 
(n = 13,142) 



Full-text articles assessed 
for eligibility 
[n = 789) 



Included {n = 270) 



Articles included in 
quantitative count 
(n = 338) 



Articles included in 
descriptive analysis 
(n = 204) 



Records excluded 
fn = 12,352) 



Full-text articles excluded, 
with reason 5 
(n=519) 
Duplicates (n-21) 
Mot CIM (n=146) 
No costs (n=186) 
Reviews (n~49) 
Ongoing studies (n=17) 

Comments (n=l9) 
Study protocols (n - 14} 

Other <n-37) 
Not in English (n=30) 



Older articles (published 
in 2000 and before) 
(n = 134) 



4 



Herman PM, Poindexter BL, Witt CM, etal. BMJ Open 201 2;2:e001 046. doi:1 0.1 1 36/bmjopen-201 2-001 046 



Economics of complementary and integrative medicine 



supplied list search for a total of 338 economic evalua- 
tions of CIM. Of these, 204 (60%) were published from 
2001 through 2010 (114 full and 90 partial economic 
evaluations). Of the recent full economic evaluations 
almost all (103, 90%) examined the effect of one CIM 
therapy and most of the balance (10, 9%) examined the 
effect of two or more CIM therapies provided by the 
same practitioner. Only one looked at the effect of mul- 
tiple CIM therapies provided by different CIM provi- 
ders. 52 CIM was generally evaluated as an adjunct to 
usual care. 

As shown in table 2, the 204 economic evaluations pub- 
lished in the past 10 years are spread across a wide range 
of CIM therapies applied to a number of different study 
populations. The biggest concentration of full economic 
evaluations (19 in number) pertained to the use of 
NCCAM's definition of manipulative and body-based prac- 
tices (eg, chiropractic, osteopathic manipulation, massage, 
etc) for the treatment of back pain. 53-72 However, even 
this subgroup is fairly heterogeneous in terms of the 



therapy (or therapies) tested and/or the type of back pain 
treated. Eight of these comparisons involved chiropractic 
care for back pain; one for chronic, 53 one for acute 57 and 
six for either type. 59 60 63 64 67 68 Five evaluated spinal 
manipulation and manual therapy provided by phy- 
siotherapists for chronic back pain (one), 65 acute back 
pain (two) 58 69 or either (two). 56 68 Four involved osteo- 
pathic manipulation; one for chronic 71 and one for sub- 
acute back pain 72 and two for musculoskeletal conditions 
including back pain. 66 68 Three evaluated massage; two for 
chronic 55 62 and one for acute back pain. 57 The last two 
studies evaluated a musculoskeletal physician (treatment 
'with a combination of manual therapy, injections, acu- 
puncture and other pain management techniques') for 
orthopaedic referrals; 54 and a Finnish folk medicine prac- 
tice called 'bone setting' for the treatment of patients with 
chronic back pain. 61 

Table 3 shows the results of the application of the 
35-item BMJ checklist to the full economic evaluations 
published 2001-2010. 41 On average, the number of 



Table 2 Types of individual complementary and integrative medicine (CIM) therapies studied for various conditions and in 


various populations: 2001-2010 (reported as the ratio of the total number of economic evaluations to the number of full 


economic evaluations) 


















Manipulative 


















and 






Other 










body-based 




Natural 


mind-body 




CIM in 


Other CIM 






practices 


Acupuncture 


products 


medicine 


Homeopathy 


general 


therapies* 


Totalst 


Back pain 


28:19 


11 :10 


2:2 




1 : 1 


3:0 


2:2 


42:29 


Rheumatic 


9:5 


6:4 


6:6 


2:2 




1 :0 


4:3 


27: 19 


disorders 


















Mixed populations 


4:1 


6:1 


2:1 


3:1 


9:5 


2:1 


3:2 


24:9 


Cardiovascular 




1 :0 


8:6 


6:4 


1 :1 




3:1 


18: 12 


disease and 


















diabetes 


















Infection (various) 






6:4 




7:4 






13:8 


Surgery 


1 : 1 


2:2 


4:3 


5:4 








12: 10 


Members of 


3:0 


2:0 






1 :0 


7:0 




12:0 


insurance plans 


















Mental disorders 




2:2 




5:3 


1 :1 


1 :0 


2:0 


11 :6 


(various) 


















Older populations 






6:3 


2:0 






3:1 


11 :4 


Headaches 


1 :0 


3:3 




4:3 


1 :1 






9:7 


Children (various 


1 :0 








6:4 


1 :0 


1 :0 


9:4 


conditions) 


















Cancer 


2:1 


2:1 


1 : 1 


2:2 




2:0 




8:4 


Pregnancy and 




5:5 


1 :0 


1 :0 








7:5 


women's health 


















Allergies 




1 : 1 






3:1 




1 : 1 


5:3 


Other conditions^ 


1 :1 


1 : 1 


3:3 


5:4 


2:1 


2:0 


6:2 


19: 11 


Totalst 


45:25 


41 :29 


38:28 


27:16 


24:13 


18:1 


25:12 


204:114 


*Other CIM therapies included aromatherapy, healing touch, Tai Chi, Alexander technique, spa therapy, music therapy, electrodermal 




screening, clinical holistic medicine, naturopathic medicine, anthroposophic medicine, water-only fasting, Ornish Program for Reversing Heart 


Disease, use of a corset and use of a traditional mental health practitioner. 










tTotals across (down) columns will not add to numbers in the totals column (row) due to individual studies addressing more than one CIM 


therapy (patients in more than one group). 














pother conditions studied included patients with multiple chemical sensitivities, respiratory disease, pharyngeal dysphagia, dyspepsia, 


functional bowel disorders, other functional disorders, venous leg ulcers, major burns and constipation; patients who rated themselves as 


physically ill or having low quality of life; patients in home hospice or with home nursing; long-term care workers and prisoners. 





Herman PM, Poindexter BL, Witt CM, etal. BMJ Open 201 2;2:e001 046. doi:1 0.1 136/bmjopen-201 2-001 046 



5 



Economics of complementary and integrative medicine 



Table 3 Comparison of various quality measures between economic evaluations of complementary and integrative medicine 
(CIM) and conventional medicine 




Economic evaluations of CIM 






Cost-utility analyses 
(CUAs) across all 
medicinet 










Higher 










All full 


2001- 


-2005 2006-2010 


quality 


CUAs 


1998-2001 


2002-2005 




n= 1 1 4 


n=oy 


n _cc 

n=oo 


n=31 


n=27 


n=ouu 


n _CQ7 

n=Do# 


Average percentage met of applicable items on 


72 


71 


73 


87 


89 






divij cnecKiist 
















Presented the study perspective clearly (%) 


61 


58 


64 


Q"7 
Of 




74 


83** 


Presented the study time horizon (%) 


96 


98* 


93* 


1 UU 


i nn* 
1 UU 


75 


87* 


Conducted and reported sensitivity analysis (%) 


32 


22** 


44** 


1 00 


no** 

yo 


93 


84** 


Discounted costs and health effects, where 


60 


25* 


76* 


y4 


1 UU 


85 


84* 


appropriate (%)§ 
















Stated year of currency for resource costs (%) 


59 


54 


60 


77 


78** 


83 


85** 


Separate reporting of resource use (trials), 


52 


51 


53 


71 


70 






parameters (models) and unit costs 
















(for transferability) 
















Disclosed funding sources (%) 


72 


58* 


76* 


84 


93* 


65J* 




Industry sponsored (%) 


10 


12 


11 


10 


7 


18* 




Average Tufts quality score (CUAs only) 










4.75*** 


4.25**** 




*% 2 Test p value<0.001 . 
**X 2 Testp value<0.01. 

***t Test p value=0.002; comparisons were made between CIM economic evaluations published 2001-2005 and those published 2006-2010, 
and between CUAs of CIM 2001-2010 and CUAs of all medicine 2002-2005. 
tData from table 3 in Neumann. 77 

j:Data from table 3 in Neumann et al. 76 Industry sponsored was calculated as a percent all studies 1976-2001 . 

§Denominators for the percentages reported in this row are the number of studies which evaluated impacts past 1 year in either the base 
case or in sensitivity analyses. For the first five columns the denominators are 25, 8, 17, 16 and 1 1 , respectively. This information was not 
available for the last two columns. 



applicable items met by each article stayed fairly con- 
stant during this period. However, the application of two 
key items (ie, the proper use of discounting and the 
inclusion of sensitivity analysis) and the disclosure of 
funding sources improved significantly, and reporting of 
the study time horizon worsened significantly. As 
expected, the average overall and individual-item per- 
centages were higher for the higher-quality articles 
(those meeting the five study-quality criteria) and for 
CUAs of CIM. It is not surprising that CUA's quality is 
higher. They generally involve more effort than other 
CEAs and are required or recommended by various 
national guidelines. 42 73-75 Nevertheless, it seems as 
though the quality of CUAs of CIM is generally compar- 
able to, or slightly better than, that seen in CUAs across 
all medicine, at least in terms of the Tufts quality score, 
disclosure of funding sources and the five items where 
comparable data are available. 76 77 

The number of full evaluations meeting each of the 
five study-quality criteria are: comparison to usual care 
97 (85%), all costs from a recognised perspective 
96 (84%), health outcomes from a randomised or matched- 
control trial 86 (75%), patient-specific data on costs and 
outcomes 105 (92%) and sensitivity analyses 37 (32%). 
Sixty-two (54%) of full evaluations met the first four of 
these and 31 (27%) met all five. A summary of the 
results of these 31 higher-quality articles (covering 28 



different studies) is shown in table 4. 54 60 62 68 71 78 " 103 
Twenty-two of these articles (19 of the studies) reported 
resource use (trials) or model parameters (models) sep- 
arate from unit prices — a minimum measure of study 
transferability 54 62 68 71 78 80 " 85 87 " 93 95 100 101 103 For 
those studies which included a randomised trial, the 
modified Jadad scores ranged from 2 to 4 on a scale 
from 0 to 4. The Tufts CEA Registry quality scores for 
the studies containing a CUA ranged from 4 to 6.5 on a 
scale from 1 to 7. The percentage of the applicable 
items on the BMJ checklist met by these studies ranged 
from 66% to 97%. 

Of the 56 comparisons made in these studies, 16 (29%) 
are cost saving — that is, the added CIM therapy had 
better health outcomes and lower costs than usual 
care alone. Cost savings were seen for acupuncture 
alone (instructional visits with an acupuncturist followed 
by home self-care by the partner for pregnant women 
with breech presentations at 33 weeks in terms of reduc- 
tions in both breech presentation at birth and ceasar- 
eans in the Netherlands, 91 and treatment by traditional 
Chinese medicine-trained licensed acupuncturists in 
private acupuncture clinics in the UK for low-back pain 
in terms of quality-adjusted life-years or QALYs from the 
societal perspective 85 ) and in combination with other 
therapies (along with manual therapy, injections and 
other pain management for patients referred to an 
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fable 4 Summary of results of complementary and integrative medicine (CIM) economic evaluations that met five study-quality criteria (31 articles representing 28 studies) 

Resource 
use (trials), 
parameters 
(models), 
and unit 
costs (both) 
reported 
separately? 



CIM therapy 
compared to 
usual care 
alone* 



Treatment 
duration/ 

study Patient 
duration population 



Primary outcome(s) 



Setting (information 
often limited by what 
was reported) 



Sample 
size 



Study 
design and 
quality 
scoresf 




Form and 
perspective 
of economic 
evaluation 



Incremental 
cost-effectiveness 
ratio (2011 US$)+ 



Acupuncture studies 



Brown 
etaP 4 



van den 
Berg et aP A 



Adjunctive 
acupuncture, 
manual therapy, 
injections and 
other pain 
management 

Adjunctive breech 
version acumoxa 



Up to 1 year/ 
1 year 



Ratcliffe 
et aP 5 and 
Thomas 
etaP 9 

Kim et a/ 81 

Witt etaP 7 
Witt etaP 6 



Adjunctive 
acupuncture 



Adjunctive 
acupuncture 



Adjunctive 
acupuncture 



Adjunctive 
acupuncture 



2 visits/from 
33 weeks to 
delivery 




3 months/ 
2 years 



10 

treatments in 

3-month 

cycles/ 

5 years 

3 months/ 

6 months 



Patients referred 
for an orthopaedic 
outpatient 
consultation who 
were classified as 
unlikely to require 
surgery 

Pregnant women 
with breech 
presentation at 
33 weeks 



Patients with 
low-back pain 



3 months/ 
6 months 



Clinical: SF-36 and, 
if appropriate, 
Aberdeen Low Back 
Pain Scale or 
Edinburgh Knee 
Function Scale; 
economic: EQ5D 
Economic: 

percentage of breech 
presentations at 
delivery — two 'main 
analyses'— with and 
without the option of 
external cephalic 
versions 

Clinical: bodily pain 
fm SF-36; economic: 
QALYs fm SF-6D 



60-year-old women 
with first time acute 
low-back pain 



Patients with 
dysmenorrhoea 



Patients with 
chronic low-back 
pain 



Clinical: 
Roland-Morris 
Disability, symptom 
bothersomeness; 
economic: QALYs fm 
literature 
Clinical: pain 
intensity VAS; 
economic: QALYs fm 
SF-6D 

Clinical: Hannover 
Functional Ability 
Questionnaire; 
economic: QALYs fm 
SF-6D 



Individualised care 829 
from one 'physical 
medicine' physician in 
a hospital outpatient 
clinic in Scotland 



2 instructional visits to NA 
an acupuncturist 
followed by daily home 
self-care, the 
Netherlands 



Up to 10 treatments 239 
from a TCM-trained 
acupuncturist in 
acupuncture clinic in 
the UK 

Hospital-based NA 
licensed oriental 
medical doctors in 
South Korea 



Up to 15 sessions with 201 
a physician trained in 
acupuncture 
(A-diploma) in 
Germany 

Up to 15 sessions with 2518 

a physician trained in 

acupuncture 

(A-diploma) in 

Germany 



R(2)81% 
BMJ 



Yes 



CEA-H 
CUA-H 



Decision tree 
model 
81% BMJ 



Yes 



R(3) 
Tufts 5 

94%/94% 

BMJ 

Markov 
model 
Tufts 4.5 
94% BMJ 



R(3) 
Tufts 5.5 
77% BMJ 



R(3) 
Tufts 4.5 
73% BMJ 



CEA-P 
CEA-P 



Yes 



Yes 



CUA-S 
CUA-P 



CUA-S 



No 



No 



CUA-S 



CUA-S 



Cost saving 
Cost saving 



Cost savings 
Cost savings 



Cost saving 
US$8755/QALY 



US$3086/QALY 



US$4708/QALY§ 



US$16230/QALY§ 
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Table 4 Continued 


















Resource 






















use (trials), 






















parameters 






















(models), 








CIM therapy 


Treatment 










Study 


and unit 


Form and 






compared to 


duration/ 






Setting (information 




design and 


costs (both) 


perspective 


Incremental 




usual care 


study 


Patient 




often limited by what 


Sample 


quality 


reported 


of economic 


cost-effectiveness 




alone* 


duration 


population 


Primary outcome(s) 


was reported) 


size 


scoresf 


separately? 


evaluation 


ratio (2011 US$)+ 


Witt et aP 9 


Adjunctive 


Up to 15 


Patients with 


Economic; QALYs 


10—15 sessions with 


3182 


R (2) 


No 


CUA-S 


US$18225/QALY§ 




acupuncture 


treatments/ 


headache 


fm SF-6D 


physician trained in 




Tufts 5.5 












3 months 






acupuncture 




88% BMJ 


















(A-diploma) in 






















Germany 












Willich 


Adjunctive 


Up to 15 


Patients with 


Clinical: Neck Pain 


10—15 sessions with 


3451 


R (2) 


No 


CUA-S 


US$19226/QALY§ 


etaP 4 


acupuncture 


treatments/ 


chronic neck pain 


and Disability Scale; 


physician trained in 




Tufts 5 












3 months 




economic; QALYs fm 


acupuncture 




88% BMJ 
















SF-6D 


(A-diploma) in 






















Germany 












WondGrling 


Adjunctive 


3 months/ 


Patients with 


Clinical: headache 


Acupuncture-trained 


401 


R (3) 


Yes 


CUA-S 


US$19785/QALY 


efa/ 100 and 


acupuncture 


1 year 


chronic headache 


severity score; 


physiotherapists in 




Tufts 5 




CUA-P 


US$21 074/QALY 


Vickers 








economic: QALYs fm 


own clinics in the UK 




97%/93% 








etaP 3 








SF-6D 






BMJ 








Reinhold 


Adjunctive 


3 months/ 


Patients with 


Economic: QALYs 


10—15 sessions with 


489 


R (3) 


No 


CUA-S 


US$27900/QALY§ 


etaP 6 


acupuncture 


3 months 


chronic hip or knee 


fm SF-6D 


physician trained in 




Tufts 4 














osteoarthritis 




acupuncture 




87% BMJ 








Witt et aP 8 










(A-diploma), Germany 












Adjunctive 


Up to 15 


Patients with 


Economic: QALYs 


10—15 sessions with 


981 


R (3) 


No 


CUA-S 


US$281 37/QALY§ 




acupuncture 


treatments/ 


allergic rhinitis 


fm SF-6D 


physician trained in 




Tufts 4 












3 months 






acupuncture 




94% BMJ 


















(A-diploma) in 






















Germany 












ManipulativG 


and body-based practices — see also Brown et al 
















Korthals-dG 


Manual therapy 


6 weeks/ 


Patients with neck 


Clinical: perceived 


Up to 6 weekly 45 min 


183 


R (3} 


Yes 


CEA-S 


Cost saving 


Bos et aP 2 




1 year 


pain 


recovery, pain VAS, 


sessions with a 




Tufts 6.5 




CEA-S 


Cost saving 










and Neck Disability 


physiotherapist who is 




83% BMJ 




CEA-S 


Cost saving 










Index; economic: All 


also a registered 








CUA-S 


Cost saving 










clinical plus QALYs 


manual therapist in the 




















fm EQ-5D 


Netherlands 












Williams 


Adjunctive 


2 months/ 


Patients with 


Clinical: Extended 


3 or 4 sessions with a 


187 


R (3) 


Yes 


CUA-P 


US$8730/QALY 


et aP 


osteopathic spinal 


6 months 


subacute (2- 


Aberdeen Spine 


general practitioner 




Tufts 5 










manipulation 




1 2 week) back 


Pain Scale; 


who is a registered 




89% BMJ 














pain 


economic: QALYs fm 


osteopath at own clinic 




















EQ-5D 


in UK 












UK BEAM 


Adjunctive spinal 


3 months/ 


Patients with 


Economic: QALYs 


8 sessions with a 


1287 


R(3) 


Yes 


CUA-P 


US$8425/QALY 


Trial 


manipulation and 


1 year 


low-back pain 


fm EQ-5D 


chiropractor, 












Team 68 


exercise 








osteopath, or 














Adjunctive spinal 








physiotherapist at a 




Tufts 6 




CUA-P 


US$10642/QALY 




manipulation 








private or NHS site in 




93% BMJ 


















the UK 
































Continued 
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CIM therapy 


Treatment 




compared to 


duration/ 




usual care 


study 


Patient 


alone* 


duration 


population 


Alexander 


6 lessons/ 


Patients with 


technique 


1 year 


chronic or 


Alexander 


6 lessons/ 


recurrent 


technique plus 


1 year 


non-specific back 


exercise^ 




pain 


Massage 


6 sessions/ 






1 year 




Massage plus 


6 sessions/ 




exercise^ 


1 year 




Treatment in a 


Unspecified/ 


Patients with acute 


chiropractic clinic 


1 year 


low-back pain 



Primary outcome(s) 



Setting (information 

often limited by what Sample 

was reported) size 



Study 
design and 
quality 
scoresf 



Resource 
use (trials), 
parameters 
(models), 
and unit 
costs (both) 
reported 
separately? 



Form and 

perspective Incremental 

of economic cost-effectiveness 

evaluation ratio (201 1 US$)t 



etaF 



Natural products 



Braga 
etaP 02 



Stevenson 
etaP 03 and 
Stevenson 
etaF 



Adjunctive 
preoperative 
arginine and co-3 
fatty acid 
supplementation 

Vitamin K1 



Trevithick 
etaF 



Schmier 
etaF 



Adjunctive 
antioxidants 
(vitamins C and E 
and (3-carotene) 

Adjunctive co-3 
fatty acid 
supplementation 



5 days/ 
5 days plus 
hospital stay 



1 0 years/ 
1 0 years 



25 years/ 
25 years 



42 months/ 
42 months 



Patients with 
chronic low-back 
pain 



Patients with 
gastrointestinal 
cancer undergoing 
surgery 



Postmenopausal 
women with 
osteoporosis/ 
osteopenia 



Clinical: 
Roland-Morris 
Disability 
Questionnaire 
(RMDQ); economic: 
above plus QALYs 
fm EQ-5D 



economic: pain 
severity 100 mm 
VAS and revised 
Oswestry Disability 
Questionnaire 

Economic: 
percentage of 
patients without 
complications 



Clinical: osteoporotic 
fracture; economic: 
QALYs fm the 
literature 



Cohort of Ontario 
residents aged 50- 
54 (prevention of 
cataracts) 

Males with a 
history of a heart 
attack 



Clinical: cataract 
formation 



Economic: fatal Mis 
and cardiovascular 
deaths 



Alexander technique 
teachers and massage 
therapists at own 
locations in the UK 



579 



Doctors of Chiropractic 1943 

in own clinics in 

Oregon, the USA 837 



12.5 g arginine, 3.3 g 204 

co-3 fatty acids and 

1.2 g RNA in liquid 

daily taken orally for 

5 days before surgery, 

Italy 

1 0 mg/day of vitamin NA 
K1 daily, the UK 



750 mg/day vitamin C, NA 
600 mg/day vitamin E 
and 1 8 mg/day 
(3-carotene daily, 
Canada 

'Fish oil pills', the USA NA 



R(3) 
Tufts 5.5 

97% BMJ 
MC 

66% BMJ 



R(3) 

88% BMJ 



Patient-level 

simulation 

model 

Tufts 4.5 

81%/84% 

BMJ 

Markov-type 
cohort model 
79% BMJ 



Decision 
analytic 
model 
77% BMJ 



Yes 



No 



No 



Yes 



Yes 



Yes 



CUA-P 
CEA-P 
CUA-P 
CEA-P 

CUA-P 
CEA-P 
CUA-P 
CEA-P 
CEA-P 

CEA-P 



CEA-H 



US$13300/QALY 
US$255/RMDQ pt 
US$12022/QALY 
US$144/RMDQ pt 

Dominated 
US$1010/RMDQ pt 
US$11959/QALY 
US$354/RMDQ pt 
US$21/pain mm 

US$0.73/pain mm 



Cost saving 



CUA-P Cost saving 



CEA-P Cost saving 



CEA-S Cost saving 



CEA-P US$11903/fatal Ml 

avoided 

Continued 
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CIM therapy 
compared to 
usual care 
alone* 



Treatment 
duration/ 
study 
duration 



Patient 
population 



Primary outcome(s) 



Setting (information 

often limited by what Sample 

was reported) size 



Study 



Resource 
use (trials), 
parameters 
(models), 
and unit 



design and costs (both) 



quality 
scorest 



reported 
separately? 



Form and 

perspective Incremental 

of economic cost-effectiveness 

evaluation ratio (201 1 US$)t 



Lamotte 
etaP 



Quilici 
etaP 



Franzosi 
etaP 



Black 

etal 78 



Adjunctive co-3 
polyunsaturated 
fatty acids 



3.5 years/ 
lifetime 



Patients after an 
acute myocardial 
infarction 



Adjunctive co-3 
polyunsaturated 
fatty acids 



Adjunctive co-3 
polyunsaturated 
fatty acids 

Adjunctive 

glucosamine 

sulphate 



4 years/ 
lifetime 



3.5 years/ 
3.5 years 



Patients after an 
acute myocardial 
infarction 



Patients with 
recent myocardial 
infarction 



22.6 years/ Patients with 
22.6 years osteoarthritis of the 
knee 



Other complementary and integrative medicine therapies 



Wilson and 
Datta 95 

Herman 
etaP 



Adjunctive 
yang-style tai chi 



Adjunctive 
naturopathic care 
including 
acupuncture, 
relaxation 
exercises, dietary 
and exercise 
advice 



1 year/1 year 



3 months/ 
6 months 



Nursing home 
residents at 
average risk for a 
fall 

Patients with 
chronic low-back 
pain 



Economic: life-years 
saved 



Economic: life-years 
gained (LYG), 
QALYs fm the 
literature, deaths 
avoided 

Clinical: death and 
non-fatal Ml or 
stroke; economic: 
LYG 

Clinical: pain, 
function, joint space 
loss; economic: 
QALYs fm the 
literature 

Economic: hip 
fractures avoided 



Clinical: Oswestry 
Disability 
Questionnaire; 
economic: QALYs fm 
SF-6D 



-465 mg EPA and NA 
-385 mg DHA ethyl 
esters in a daily 
gelcap, Australia, 
Belgium, Canada, 
Germany and Poland 



-465 mg EPA and NA 
-385 mg DHA ethyl 
esters in a daily 
gelcap, the UK 

-465 mg EPA and 5664 
-385 mg DHA ethyl 
esters in a daily 
gelcap, Italy 

Glucosamine sulphate NA 
powder 1500 mg daily 
in oral solution, the UK 



2 classes/week NA 
monitored by a 
certified tai chi 
instructor and an 
assistant, the USA 
Twice weekly visits to 70 
licensed naturopathic 
doctors also trained in 
acupuncture in a 
worksite clinic in 
Canada 



Decision tree Yes 

model 

89% BMJ 



Markov Yes 
model 
Tufts 5 
93% BMJ 

R (4) No 

85% BMJ 

Cohort Yes 
simulation 
model 
84% BMJ 



Decision tree Yes 

model 

96% BMJ 



R(3) 
Tufts 5 
96% BMJ 



CEA -P 
CEA -P 
CEA -P 
CEA -P 
CEA -P 
CEA -P 
CUA-P 

CEA-P 
CUA-P 



Yes 



CEA-P 

CUA-S 
CEA-E 
CBA-E 



US$541 3/LYG 

Australia 

US$81 84/LYG 

Belgium 

US$4476/LYG 

Canada 

US$6750/LYG 

Germany 

US$7747/LYG 

Poland 

US$28420/LYG 
US$35940/QALY 



US$41 867/LYG 



US$59053/QALY 



Cost saving 



Cost saving 

US$191/absentee 
day avoided 
Cost saving 
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Treatment 
duration/ 
study Patient 
duration population 



Primary outcome(s) 



Setting (information 
often limited by what 
was reported) 



Sample 
size 



Study 
design and 
quality 
scorest 



Resource 
use (trials), 
parameters 
(models), 
and unit 
costs (both) 
reported 
separately? 




Form and 




perspective 


Incremental 


of economic 


cost-effectiveness 


evaluation 


ratio (2011 US$)+ 


CEA-S 


US$21 59/BASFI pt 




(spa in Austria) 


CEA-S 


US$421 5/BASFI pt 




(spa in the 




Netherlands) 


CUA-S 


US$12703/QALY 




(spa in Austria) 


CUA-S 


US$31 609/QALY 




(spa in the 




Netherlands) 


CUA-S 


US$46443/QALY 




(VAS) 


CUA-S 


US$92886/QALY 




(SF-6D) 



Van 

Tubergen 




Zijlstra 
efa/ 101 



Combined 

spa-exercise 

therapy 



3 weeks/ 
40 weeks 



Patients with 

ankylosing 

spondylitis 



Adjunctive spa 
therapy 



2.5 weeks/ 
1 year 



Patients with 
fibromyalgia 



Clinical: Bath 
Ankylosing 
Spondylitis 
Functional Index 
(BASFI 10pts), pain 
VAS, well-being VAS 
and morning 
stiffness in minutes; 
economic: above 
plus QALYs fm 
EQ-5D 

Economic: QALYs 
fm VAS and SF-6D 



3-week stay at one of 
two spa-resorts with 
therapy provided by 
trained 

physiotherapists, the 
Netherlands 



120 



R (3) Yes 
Tufts 4.5 

90% BMJ 



18-day stay at a spa in 
Tunisia with a variety 
of treatments, the 
Netherlands 



128 



R(3) 
Tufts 4 
97% BMJ 



Yes 



*The use of the term 'adjunctive' in this column indicates complementary and alternative medicine (CAM) therapies used in addition to usual care for that condition unless otherwise indicated. 
tStudy design: R, randomised; MC, matched controls and/or results statistically adjusted for baseline differences. A modified Jadad score (maximum score = 4) is provided if the study was 
randomised. If the study was a CUA and a quality score was available from the Tufts Medical Center Institute for Clinical Research and Health Policy Studies CEA Registry (https://research. 
tufts-nemc.org/cear/Default.aspx), it is reported. Quality scores range from 1 to 7 with 7 representing the highest quality. The last number is the percent of the applicable items on the BMJ 
35-item quality checklist that this study met. If a study had more than one publication, both percentages were reported. The BMJ checklist is found in Drummond efa/. 41 
JThe costs reported in each study were first converted to US$ using the Federal Reserve annual exchange rate (http://www.federalreserve.gov/releases/g5a/20090102/, accessed 30 Jan 2012) 
for the study's currency year and then inflated to 201 1 values using the medical care component of the Consumer Price Index (http://www.bls.gOv/cpi/cpi_dr.htm#2007, accessed 30 Jan 2012). 
In comparisons labelled as cost saving the CIM therapy both improved health and lowered costs compared to usual care. In the comparison labelled dominated the CIM therapy had worse 
health outcomes and higher costs than usual care. 

§These studies did not report a currency year so it was estimated as being 1 year prior to publication. 
HCompared to usual care plus exercise. 

CBA, cost-benefit analysis; CEA, cost-effectiveness analysis; CUA, cost-utility analysis; DHA, Docosahexaenoic acid; E, employer perspective; EPA, Eicosapentaenoic acid; H, hospital 
perspective; Ml, myocardial infarction; P, payer perspective; QALY, quality-adjusted life-year; S, societal perspective; VAS, visual analogue scale. 



Economics of complementary and integrative medicine 



orthopaedic surgeon's office in Scotland who were 
unlikely to need surgery in terms of both improvements 
in health-related quality of life and QALYs 54 ) . Cost 
savings were also seen for manual therapy delivered by a 
physiotherapist, who is also a registered manual therapist, 
for neck pain in terms of perceived recovery, pain, neck 
disability and QALYs 82 ; for preoperative oral supplemen- 
tation with arginine and co-3 fatty acids for patients with 
gastrointestinal cancer undergoing surgery 102 ; for 
vitamin K x supplementation for postmenopausal women 
with osteopenia and osteoporosis in terms of QALYs 103 ; 
for supplementation with vitamins C and E and 
(3-carotene for cataract prevention 90 ; for fish oil supple- 
mentation in men with a history of heart attack 87 ; for tai 
chi to prevent hip fractures in nursing home residents 95 
and for naturopathic care offered through a worksite 
clinic for chronic low-back pain in terms of both reduc- 
tions in absenteeism and gains in QALYs. 80 

Of the 28 cost-utility comparisons, one (massage for 
low-back pain 62 ) was dominated — that is, had worse 
health outcomes and higher costs than usual care. 
Five (18%) are cost saving, 54 80 82 85 103 5 (18%) have 
incremental cost-effectiveness ratios (ICERs) between 
US$0 and US$10 000 per quality-adjusted life-year 
(QALY), 68 71 81 85 97 and 89% had ICERs less than 
US$50 000/QALY; a threshold often considered to repre- 
sent the upper limit of society's value for a QALY 104 The 
cost-saving cost-utility studies were included in the para- 
graph above (ie, those that mention QALYs). The 
studies with cost-utility ICERs between US$0 and 
US$10 000 per QALY were: treatment by traditional 
Chinese medicine-trained licensed acupuncturists in 
private acupuncture clinics in the UK for low-back 
pain. 85 hospital-based acupuncture by licensed oriental 
medical doctors in South Korea for 60-year-old women 
with first-time acute low-back pain, 81 acupuncture from 
physicians with at least 140 h of training (A-diploma) in 
Germany for patients with dysmenorrhoea, 97 osteopathic 
spinal manipulation by a general practitioner who is a 
registered osteopath in the UK for patients with sub- 
acute back pain, 71 and an exercise programme plus 
spinal manipulation from a chiropractor, osteopath or 
physiotherapist at a private or National Health Service 
(NHS) site in the UK for low-back pain. 68 The average 
percentage of applicable BMJ checklist items met by 
each study was slightly lower for those studies with at 
least one cost-saving comparison (85% vs 88%), but the 
difference was not statistically significant (t test=0.75, p 
value=0.460). 

DISCUSSION 

This comprehensive systematic review identified 338 eco- 
nomic evaluations of CIM; 204 of which were published 
recently (2001-2010) covering a wide range of CIM ther- 
apies for a variety of populations. Although most patients 
who use CIM use more than one modality 35 and despite 
the attention given to integrative medicine (coordinated 



access to conventional medicine and CIM) , this system- 
atic review found only one study that examined the 
effects of use of multiple CIM practitioners. 52 In general, 
the quality of the recent full economic evaluations has 
held constant and is in line with what is seen in economic 
evaluations in conventional medicine. Details of the 31 
recent higher-quality full economic evaluations indicate 
potential cost-effectiveness and cost savings across a 
variety of CIM therapies applied to different conditions. 
Owing to the non-generalisable nature of economic eva- 
luations, the cost estimates shown are specific to their 
study settings. 40 However, 22 articles provided at least 
the minimum information for study transferability. 
Therefore, their results could be adapted via modelling 
to determine the economic impact of these interventions 
in other settings. 

The strengths of this study are the comprehensive 
search strategy, which revealed a substantial number of 
published economic evaluations of CIM, the use of two 
reviewers and the use of multiple measures of study 
quality. Higher-quality studies were identified and high- 
lighted for policy makers using a simple objective list of 
quality criteria, which reduced the potential for bias. 
The weaknesses of this study are similar to those of the 
other systematic reviews. The reviewers were not blinded 
to journals and article authors, which may have influ- 
enced results. Also, some aspects of what makes a quality 
economic evaluation could not be judged from what was 
reported. For example, ideally, pragmatic trials enrol 
patients typical of normal caseload in typical settings 
with typically trained and experienced practitioners fol- 
lowing them under routine conditions (ref. 37 , p. 251). 
Judgements as to whether these criteria were met were 
not always possible from the reports, and were beyond 
the scope of this review. Finally, publication bias was not 
assessed. However, since the major goal of this study was 
to establish the extent of the published literature on this 
topic and to highlight the results of the higher-quality 
studies, it is not clear that publication bias is relevant 
here. 

The number of economic evaluations of CIM found 
and reviewed by this study far exceeds the numbers 
found in previous studies. This study found a 

total of 338 economic evaluations of CIM published 
between and including 1979 and 2010; 211 of these were 
full economic evaluations. White and Ernst 19 identified 
34 economic evaluations of CAM published 1987-1999; 
11 of which were full economic evaluations. Between 
1999 and October 2004, Herman et aP identified 56 eco- 
nomic evaluations of CAM (39 full evaluations). Between 
1994 and May 2004 Hulme and Long 21 identified 19 full 
economic evaluations of CAM, and over a similar period 
(1995-2007) Doran et aP found 43 economic evalua- 
tions (15 full evaluations). Maxion-Bergemann et al 11 
identified 5 (1 full) economic evaluations over an 
unspecified search period. The large number of eco- 
nomic evaluations found in this study reflects the facts 
that: (1) all evaluations from previous reviews were 
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included; (2) a number of studies have been published 
since the last search dates of prior reviews and (3) a more 
extensive search strategy was used. It should be noted 
that 20% of the articles (68 of 338) in this review were 
identified through bibliography searches and from 
expert lists. Therefore, even the application of a long list 
of search terms to multiple databases does not guarantee 
that all CIM studies will be identified. However, there is 
some evidence that the indexing of these articles in 
medical databases is improving; studies from bibliograph- 
ies and expert lists made up 32% of found articles pub- 
lished 2000 and before, but only 12% recent articles. 

There are several implications of this study for policy 
makers, clinicians and future researchers. First, there is 
a large and growing literature of quality economic eva- 
luations in CIM. However, although indexing is improv- 
ing in databases, finding these studies can require going 
beyond simple CIM-related search terms. Second, the 
results of the higher-quality studies indicate a number of 
highly cost-effective, and even cost saving, CIM therap- 
ies. Almost 30% of the 56 cost-effectiveness, cost-utility 
and cost-benefit comparisons shown in table 4 (18% of 
the CUA comparisons) were cost saving. Compare this 
to 9% of 1433 CUA comparisons found to be cost saving 
in a large review of economic evaluations across all 
medicine. 106 Third, by meeting the five study-quality cri- 
teria, the studies shown in table 4 can each be consid- 
ered a reasonable indicator of the health and economic 
impacts of the CIM therapy studied, at least in that popu- 
lation and setting. These studies, especially those showing 
cost savings, should be considered further for applicabil- 
ity in other settings. This requires the study to be transfer- 
able. 39 Fortunately, the majority of the higher-quality 
studies met our measure of study transferability — 
resource use or model parameters, and unit costs were 
reported separately. 

Given the substantial number of economic evaluations 
of CIM found in this comprehensive review, even 
though it can always be said that more studies are needed, 
what is actually needed are better-quality studies — both 
in terms of better study quality (to increase the validity 
of the results for its targeted population and setting) 
and better transferability (to increase the usefulness of 
these results to other decision makers in other settings). 
Therefore, the following recommendations are made. 

1. That all studies measuring the effectiveness of CIM at 
least consider also measuring input costs and eco- 
nomic outcomes. 

2. That at least one arm of the study be some version of 
commonly available (usual) care, and that usual care 
and all interventions studied be described in suffi- 
cient detail that decision makers in other settings can 
determine what was done and whether the study's 
usual care comparator is applicable in their setting. 

3. That consideration be given to how CIM is typically 
used (eg, multiple CIM therapies) or can be used 
(eg, coordinated integrative care models) when 
designing studies. 
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4. That changes in resource use be reported separately 
from unit costs in economic evaluations alongside 
clinical trials and that model parameters and unit 
costs be clearly reported in decision-analytic model- 
ling studies. 

5. That all economic evaluations contain sensitivity ana- 
lyses to increase the reliability of results. 

6. That more consideration be given to modelling as a 
method to estimate economic outcomes for existing 
effectiveness trial results, and to generalise existing 
quality economic evaluation results to other 
jurisdictions. 
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